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SUMMARY 

The  U  S.  Army  Engineer  Topographic  Laboratories  contracted  vilh  Software  A&E. 
Inc.  to  assist  in  conducting  applied  research  in  terrain  feature  extraction  from  synthetic 
aperture  radar  (SAR)  images.  The  purpose  of  the  research,  carried  out  in  1984,  was  to 
evaluate  the  efficacy  of  artificial  intelligence  (AI)  techniques  in  the  classification  of 
ambiguous  areas  in  SAR  images. 

Heuristics  and  standard  image  processing  techniques  were  incorporated  into  a 
knowledge-based  expert  system  which  was  then  applied  to  the  SAR  images.  Results  of  the 
application  of  these  techniques  were  compared  to  the  results  from  traditional  image 
processing  techniques.  Results  of  these  experiments  showed  that  artificial  intelligence 
techniques  hold  significant  promise  for  improving  the  understanding  of  ambiguous 
areas  in  SAR  images. 

The  success  of  the  image  understanding  experiments  led  to  the  formulation  of  a 
conceptual  model  of  a  system  that  could  be  used  as  a  test  bed  for  image  understanding 
research.  The  conceptual  model  relies  on  techniques  from  the  world  of  artificial 
intelligence  as  well  as  traditional  image  processing.  The  system  architecture  is 
sufficiently  flexible  to  allow  growth  into  an  operational  system  capable  of  supporting 
tactical  and  strategic  intelligence  applications. 
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PREFACE 

The  work  described  in  this  report  was  undertaken  as  part  of  a  series  of  projects 
performed  by  Software  A&E  in  the  application  of  artificial  intelligence  methods  to 
problems  faced  by  the  U  S.  Army  Engineer  Topographic  Laboratories  (ETL).  The  authors 
would  like  to  acknowledge  the  contributions  of  Dr.  P f .  Chen.  Mr.  Richard  Hevenor  and 
Mr.  Neil  Fox  of  ETL.  Dr.  Peter  Lemkin  provided  invaluable  insights  and  advice  on  image 
processing  techniques. 


-2  - 


ETL-0387 
SAE-DC-83-R-004 
February  1983 


1.  INTRODUCTION 


The  U  S.  Army  Engineer  Topographic  Laboratories  (ETL)  have  been  conducting 
applied  research  in  the  area  of  terrain  feature  extraction  from  SAR  images.  The  work 
has  focused  on  human  analysis  of  terrain  features  and  the  experimental  application  of 
image  processing  algorithms  to  that  analysis.  Even  though  the  experimental  results 
have  been  encouraging,  it  was  felt  that  heuristic  reasoning  from  the  realm  of  artificial 
intelligence  used  in  a  complementary  manner  with  the  image  processing  algorithms 
could  result  in  more  powerful  SAR  image-understanding  automated  systems.  Therefore, 
ETL  contracted  with  Software  A&E  to  analyze  existing  algorithms  to  determine  how  AI 
techniques  could  be  employed  to  address  problem  areas  and  to  perform  experiments 
illustrating  the  application  of  AI  to  the  image-understanding  problem. 

The  statement  of  work  specified  four  tasks: 

Task  1  Study  of  Algorithms  and  Heuristics 
Task  2  Develop  a  Skeletal  Conceptual  Model 
Task  3  Heuristics  Integration  Experiment 
Task  4  Experiment  Evaluation 

This  final  report  describes  the  tasks  carried  out  by  Software  A&E  knowledge 
engineers  in  support  of  ETL  objectives,  the  results  achieved,  and  recommendations  for 
further  development  of  systems  for  research  into  image  understanding. 
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2.  TASK  DEFINITIONS 

The  original  tasks  specified  for  the  project  were  modified  by  the  principal 
investigator  with  the  concurrence  of  Software  A&E  to  more  accurately  reflect  the 
interests  of  the  ETL  researchers.  The  tasks  actually  carried  out  during  the  course  of  the 
project  are  described  below. 

2.1  Taskl:  Study  of  Algorithms  and  Heuristics 

In  order  to  accomplish  this  task,  the  incorporation  of  existing 
image-understanding  software  from  ETL  with  the  Software  A&E  knowledge  Engineering 
System  (EES)  was  necessary.  This  led  to  the  transportation  of  ETL's  software  from  the  ETL 
HP1000  to  Software  A&E's  VAX  11/750.  Since  the  source  code  was  written  in  a  standardized 
version  of  FORTRAN  it  was  theoretically  capable  of  being  ported  to  another  system.  The 
first  task  was  accomplished  after  the  porting  of  the  EH  image  understanding  software  to 
the  VAX  11/730. 

2.2  Task  2:  Develop  a  Skeletal  Conceptual  Model 

A  conceptual  model  for  a  radar  image-understanding  system  embodying  standard 
image  processing  techniques  and  A1  heuristic  techniques  was  outlined.  It  generalized 
the  findings  of  task  3-  The  skeletal  model  was  documented  in  a  technical  memorandum 
that  is  attached  as  Appendix  B  of  this  report. 

2.3  Task  3:  Heuristics  Integration  Experiment 

In  this  task  the  development  of  the  original  Radar  Image  Classification  Aid  (RICA) 
was  initiated.  Necessary  heuristics  and  other  AI  techniques  were  integrated  into  the 
RICA  together  with  the  original  image-understanding  software  from  ETL.  Experiments 
were  performed  using  SAR  imagery  supplied  by  ETL  as  input  for  the  RICA.  A  skeletal 
conceptual  model  was  developed  in  Task  2  to  generalize  the  original  RICA.  Development 
of  the  techniques  in  the  context  of  the  new  RICA  architecture  was  completed  but 
experimental  design  and  execution  was  not  possible  due  to  the  ftinding  limitation. 
Information  gathered  during  the  software  development  phase  of  the  contract  showed 
that  the  techniques  implemented  hold  promise  for  significant  improvements  in  the 
dassificaiton  of  SAR  images. 

2.4  Task  4:  Experiment  Evaluation 

The  developments  in  Task  3  were  evaluated  and  conclusions  were  made  regarding 
the  value  of  continuing  the  research  begun  in  this  exploratory  study.  The  conclusions 
are  documented  in  this  final  report  of  the  project. 
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3-  RADAS  IMAGE  CLASSIFICATION  AID  (SICA) 

The  Radar  Image  Classification  Aid  is  an  application  of  Artificial  Intelligence 
techniques  to  the  area  of  image  processing.  ETL  currently  uses  a  Bayes  Classifier 
Program  to  identify  32  z  32  pixel  areas  called  windows  in  a  312  z  312  pixel  SAR  image. 
This  program  classifies  each  vindov  as  one  of  the  following  feature  classes:  field,  water, 
forest,  or  city.  The  result  of  this  program  is  a  16  x  16  Feature  Matrix  that  contains  the 
classification  of  each  window  in  the  SAR  image. 

Problems  arose  when  a  window  contained  a  border  between  two  or  more  feature 
classes.  As  a  partial  solution  to  this  problem  ETL  added  a  new  feature  class,  border  type,  to 
the  Bayes  Classifier  Program.  This  feature  class  was  given  to  any  window  containing  a 
border.  The  aim  of  RICA  is  to  determine  the  feature  classes  that  compose  each  of  these 
border  type  windows. 

3-1  AI  lietholodogies 

RICA  incorporated  three  applications  of  artificial  intelligence  to  the 
understanding  of  SAR  images 

•  heuristic  border  search 

•  q-NN  classification  rule 

•  frame  based  reclassification 

Each  of  these  metholodogies  are  described  in  the  following  paragraphs. 

3-1-1  Heuristic  Border  Search 

The  heuristic  border  search  technique  locates  and  arranges  all  of  the  border  type 
windows  in  the  Feature  Matrix  that  will  be  reclassified  by  the  Reclassification  System 
Module  (see  description  of  software  modules  in  appendix  A). 

The  heuristic  assumes  that  it  is  easier  to  reclassify  those  windows  that  have  the 
most  known  about  their  neighborhoods  and  harder  to  reclassify  windows  with  many 
unknowns  in  their  neighborhoods.  Note  that  if  the  method  is  provided  the  ability  to 
learn  by  updating  the  Feature  Matrix  each  time  a  border  type  window  has  been 
reclassified  then  those  border  type  windows  that  originally  had  many  unknowns  in  their 
neighborhoods  will  have  been  updated  enough  to  reclassify  them  on  subsequent 
iterations  (see  paragraph  31 .4). 

The  ordered  border  information  is  provided  to  the  Frame  Based  Reclassification 
System  for  use  in  its  consideration  of  possible  border  reclassifications. 

3-1-2  q-NN  Classification  Rule 

The  q-NN  Classification  Rule  method  examines  the  immediate  neighborhood  of  a 
given  border  type  window  and  calculates  the  percentage  that  each  of  the  five  feature 
classes  composes  in  that  neighborhood.  The  heuristic  incorporated  with  the  q-NN 
Classification  Rule  is.  "the  feature  class  with  the  highest  percentage  will  have  a  higher 
liklihood  of  composing  part  of  the  border  type  window*.  The  information  is  made 
available  to  the  Frame  Based  Reclassification  System  for  use  in  deciding  the  make-up  of  a 
border  window 
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3-1-3  Frtae  Based  Be classification 

The  Frame  Based  Reclassification  method  uses  the  information  provided  by  the 
q-NN  classification  rule  and  other  traditional  image  processing  techniques  to  decide  vhat 
combination  of  these  feature  classes  best  fits  the  information  received  about  the  vindov 
being  reclassified. 

The  decision-making  technique  is  based  on  minimal  set  cover  theory.  The 
Knowledge  Engineering  System  (KES).  a  knowledge- based  expert  system  building  tool 
available  from  Software  Architecture  and  Engineering.  Inc.,  provides  the  Hypothesize 
and  Test  (HT)  inferencing  technique  that  has  been  used  to  implement  frame  based 
reclassification. 

3.2  Results 

The  Radar  Image  Classification  Aid  correctly  reclassified  approximately  90%  of  the 
border  type  windows.  Appendix  A  contains  the  technical  memorandum  that  describes  the 
RICA  task.  It  details  the  software  created  for  the  experiment  along  with  the  results  of  the 
investigation  made  with  RICA. 


-6  - 


ETL-0387 
SAE-DC-83-R-004 
February  1983 


4.  CONCEPTUAL  MODEL 

The  RICA  system  was  derived  from  the  idea  of  a  conceptual  model  for  an 
image-understanding  system.  The  conceptual  model  is  intended  for  the  image 
understanding  researcher  and.  ultimately,  for  tactical  and  strategic  intelligence 
personnel.  The  model  combines  standard  image  processing  and  artificial  intelligence 
techniques  in  a  single  system  whose  architecture  consists  of  a  hierarchy  of  expert 
systems 

The  system  architecture  described  in  the  conceptual  model  is  extremely  flexible.  It 
allows  for  the  convenient  introduction  of  new  techniques  as  they  are  conceived.  An 
mportant  aspect  of  the  model  is  its  intelligent  front  end.  The  intelligent  front  end 
provides  the  user  with  the  flexibility  either  to  choose  the  way  techniques  are  combined 
or  to  allow  the  system  to  determine  the  best'  combination  of  image  processing  and  AI 
techniques  to  apply  to  the  task  of  understanding  a  SAR  image. 

The  architecture  of  the  model  is  described  in  detail  in  Appendix  B. 
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5  FOBTHEB  DEVELOPMENT  OF  SICA 

The  third  task  was  concerned  with  the  development  and  evaluation  of  techniques 
that  might  improve  the  classification  of  SAR  images.  Specifically,  four  enhancements 
were  developed  within  the  context  of  the  RICA  system.  These  enhancements  are 
described  below.  See  Appendix  B  for  a  full  discussion  of  the  part  that  these 
enhancements  play  in  the  conceptual  model. 

3-1  Enhancements 

5.1.1  System  Control.  The  system  control  expert  system  is  intended  to  be  a 
sophisticated  intelligent  front  end  for  an  image  understanding  system.  Among  other 
features  of  the  modelled  version,  the  front  end  would  support  the  choice  of  image 
processing  and  AI  techniques  to  be  used  in  the  task  of  understanding  a  SAR  image.  The 
fully  implemented  version  is  intended  to  make  these  choices  vith  or  without  human 
intervention,  relying  on  information  in  its  ovn  knowledge  base(s). 

For  purposes  of  the  third  task,  a  simpler  version  of  the  system  control  expert 
system  was  implemented.  This  version  provides  the  experimenter  with  the  ability  to 
manually  specify  the  combination  of  techniques  to  be  used  in  analyzing  a  SAR  image. 

5.1.2  Analysis  of  Window  Quadrants.  This  enhancement  enlarges  upon  the  pixel 
level  information  available  for  image  reasoning.  The  technique  divides  a  border  type 
window  into  quadrants.  The  area  surrounding  each  quadrant  is  then  analyzed  to 
determine  the  most  reasonable  classification  for  that  portion  of  the  border  window. 
Thus,  each  border  window  quadrant  is  analyzed  in  the  context  of  its  surrounding 
neighborhood. 

5-1.3  Neighborhood  Expansion.  This  enhancement  provides  a  larger  neighborhood 
to  the  q-NN  Classification  Rule  module  when  the  initial  3i3  neighborhood  provides 
insufficient  information  for  reasonable  classification.  This  technique  expands  the 
neighborhood  in  steps  (5x5. 7x7.  etc.)  until  sufficient  information  has  been  obtained. 

5.1.4  Learning.  When  a  border  window  has  been  reclassified,  the  learning 
enhancement  immediately  updates  the  Feature  Matrix  so  that  the  information  is  available 
during  the  reclassification  of  neighboring  windows. 

5.2  Experience  With  Enhancements 

The  enhancements  were  designed,  coded,  and  tested  during  the  final  stages  of  the 
contract.  Several  test  runs  were  executed  for  purposes  of  demonstrating  the  correctness 
of  the  code  and  to  do  preliminary  evaluation  of  the  enhancements.  The  preliminary 
evaluations  did  indicate  a  definite  improvement  in  classification  performance  that  would 
warrant  further  investigation.  Limitations  on  contract  funding  prevented  a  substantial 
number  of  experimental  runs  that  would  determine  the  relative  efficiencies  of  the 
enhancements,  both  individually  and  in  combination. 
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6.  CONCLUSIONS 

The  image  understanding  project  has  been  a  fruitful  line  of  research  that  has 
proven  the  utility  of  the  application  of  artificial  intelligence  to  image  understanding 
problems.  The  artificial  intelligence  methodologies  have  significantly  improved  the 
interpretation  and  classification  of  information  from  SAR  images. 

The  conceptual  model  of  an  image  understanding  system,  consisting  of  a  hierarchy 
of  cooperating  expert  systems,  has  been  shovn  to  be  practical  using  existing  technology. 
The  architecture  of  the  open  ended  system  exemplified  by  the  model  is  sufficiently 
flexible  to  support  further  research  and  to  provide  a  basis  for  a  production  system  for 
operational  intelligence  applications. 
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7.  RECOMMENDATIONS 

The  project  has  proven  the  practicality  and  utility  of  the  application  of  artificial 
intelligence  techniques  in  the  task  of  classifying  SAR  images.  The  research  begun  vilh 
this  project  should  be  carried  further.  Three  lines  of  research  should  be  investigated. 

First,  a  series  of  experiments  should  be  devised  and  carried  out  to  determine  the 
efficacy  of  the  various  techniques  already  implemented  in  the  RICA  system.  Information 
should  be  developed  to  show  the  performance  of  each  technique,  both  alone  and  in 
various  combinations  with  other  techniques.  The  sensitivity  of  each  technique  and 
combination  of  techniques  to  variations  in  quality  of  SAR  images  should  also  be  explored. 

Second,  the  techniques  pioneered  in  this  project  should  be  refined  and  extended. 
New  techniques  should  be  devised  and  tested  for  their  practicality  and  utility. 

Third,  the  system  embodied  in  the  conceptual  model  should  be  completed  and 
studied.  This  system  has  important  contributions  to  make  in  both  the  short  and  long 
terms.  In  the  short  term,  the  model  provides  a  flexible  and  powerful  test  bed  for  nev 
ideas  in  image  understanding.  In  the  long  term,  the  model  provides  the  basis  for  the 
economical  development  of  an  operational  image  understanding  system  of  unparalleled 
capability.  It  is  strongly  recommended  that  a  prototype  of  such  an  operational  system  be 
developed  and  evaluated. 
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APPENDIX  A 

Radar  Image  Gassification  Aid  (RICA) 
Experiment  Evaluation 


Experiment  Evaluation 
Analysis  of  the  Results  from  RICA 
Problem  Statement  1 


1 .  Introduction 

The  Radar  Image  Classification  Aid  (RICA)  is  an  application  j 

of  Artificial  Intelligence  (AI)  techniques  to  the  area  of  image 
processing.  The  U.S.  Army  Engineer  Topographic  Laboratories 

( ETL )  currently  uses  a  Bayes  Classifier  Program  to  identify  32  x 
32  pixel  areas  called  windows,  in  a  512  x  512  pixel  SAR  image. 

This  program  classifies  each  window  as  one  of  the  following 

feature  classes:  field,  water,  forest,  or  city.  The  result  of  j 

this  program  is  a  16  x  16  Feature  Matrix  that  contains  the 

classification  of  each  window  in  the  SAR  image.  Problems  arose 

when  a  window  contained  a  border  between  two  or  more  feature 

classes.  Toward  a  solution  of  this  problem  ETL  added  a  new 

feature  class?  border  type  or  BDR,  to  the  Bayes  Classifier 

Program.  This  feature  class  was  given  to  any  window  containing  a  jj 

border.  The  aim  of  RICA  is  to  determine  the  feature  classes  that 

compose  each  of  these  border  type  windows. 


2.  Primary  Applications  of  AI 

I 

There  are  three  primary  applications  of  AI  in  the  current 
RICA  system:  Heuristic  Border  Search,  q-NN  Classification  Rule, 
and  a  Frame  Based  Re-classification  System.  Each  of  these 
applications  is  represented  by  a  module  in  the  RICA  system’s 
module  hierarchy  as  shown  in  the  Appendix. 

! 

2.1  Heuristic  Border  Search 

The  purpose  of  the  Heuristic  Border  Search  Module  is  to 
locate  and  arrange  all  of  the  border  type  windows  in  the  Feature 
Matrix  that  will  be  re-classified  by  the  Re-classification  System 
Module.  The  heuristic  used  to  arrange  the  border  type  windows  is  I 

that  it  would  be  easier  to  re-classify  those  windows  that  have 
the  most  known  about  their  neighborhoods  first,  and  harder  to 
re-classify  windows  with  many  unknowns  in  their  neighborhoods. 

This  is  done  by  examining  the  immediate  neighborhood  (a  3  x  3 

matrix)  and  determining  the  number  of  unknown  or  border  type 

windows  that  reside  there.  The  idea  behind  this  is  that  when  I 

RICA  is  given  the  capability  to  'learn',  updating  the  Feature 

Matrix  each  time  a  border  type  window  has  been  re-classified, 

then  eventually  those  border  type  windows  that  originally  had 

many  unknowns  in  their  neighborhoods  will  have  been  updated 

enough  to  re-classify  them. 


2.2  q-NN  Classification  Rule 


The  q-NN  Classification  Rule  Module  is  based  on  the  "nearest 
neighbor"  principle  and  q-NN  classification  rule  [TOU74].  The 
difference  is  that  the  q-NN  Classification  Rule  Module  does  not 
actually  re-classify  a  border  type  window  in  the  Feature  Matrix. 

It  merely  examines  the  immediate  neighborhood  of  a  given  border 
type  window  and  calculates  the  percentage  that  each  of  the  five 
feature  classes  composes  in  that  neighborhood.  The  Frame  Based 
Re-classification  System  will  use  the  information  gathered  by  the 
q-NN  Classification  Rule  Module  to  make  a  decision  about  what 
feature  classes  make  up  the  components  of  the  border  window. 

2.3  Frame  Based  Re-classification  System 

The  Frame  Based  Re-classification  System  is  written  in  the 
Software  Architecture  and  Engineering's  (SAE)  Knowledge 
Engineering  System  (KES),  Hypothesize  and  Test  (HT)  subsystem 
language  [KES84].  This  RICA  module  does  the  actual 

re-classification  of  the  border  type  windows.  It  gathers 

information  about  a  given  border  type  window  and  its  surrounding 
neighborhood  from  other  RICA  modules.  That  information  is  then 
used  with  the  frame  based  descriptions  of  each  of  the  five 
feature  classes  (field,  water,  forest,  city,  and  border)  to 
decide  what  combination  of  these  feature  classes  best  fits  the 
information  received  about  the  window  being  re-classified.  The 
concept  behind  the  KES  HT  subsystem  is  based  on  minimal  set  cover 
theory  [REG83]. 

3 .  Results 

The  Initial  Classification  Module  is  the  Bayes  Classifier 
Program  from  ETL .  It  provides  the  initial  Feature  Matrix  upon 

which  the  remainder  of  the  RICA  system  is  based.  The  following 
is  a  discussion  of  the  results  of  applying  the  AI  applications  to 
the  task  of  determining  the  components  of  each  previously  defined 
border  type  window.  (The  percentages  used  in  this  report  are  the 
averages  between  the  results  of  the  images  UNF014  and  UNF026, 
shown  at  the  end  of  this  Appendix.) 

3.1  Initial  Feature  Matrix  Classification 

In  transferring  the  Initial  Classification  software  from  the 
ETL  HP  1000  to  Software  A&E's  VAX  11/750,  differences  occurred 

between  the  classification  of  the  SAR  image  obtained  at  each 

installation.  In  the  testing  of  RICA,  94.5  percent  of  the 

windows  were  classified  with  the  same  feature  classes  at  Software 
A&E  and  ETL.  Of  the  5.5  percent  that  were  classified 
differently,  approximately  2.2  percent  were  more  accurate  than 
the  classifications  obtained  at  ETL  when  the  actual  SAR  image  was  ^ 

viewed.  The  remaining  3.3  percent  were  not  classified  more 
accurately. 


The  causes  of  the  differences  in  the  initial  classifications 
have  not  been  completely  determined,  but  it  is  thought  that  they 
were  caused  by  differences  in  the  inverted  mean  and  covariance 
matrices  and/or  the  differences  in  the  precision  in  the  different 
computer  systems  that  were  used. 

Although  these  differences  may  have  had  a  slight  impact  on 
the  final  re-classification  of  the  border  type  windows,  the 
integrity  of  the  RICA  system  was  not  affected  since  the  purpose 
of  this  project  was  to  determine  whether  AI  techniques  could  aid 
in  image  understanding  and  not  actual  image  processing. 

3.2  Feature  Matrix  Re-Classification 

The  Feature  Matrix  was  composed  of  approximately  11  percent 
border  type  windows.  Of  those  windows,  RICA  re-classified  89.5 
percent  with  a  total  or  partially  correct  feature  class  value. 
The  results  from  RICA  were  compared  against  values  that  were 
determined  beforehand  by  studying  the  actual  SAR  image. 

Out  of  the  89.5  percent,  60.5  percent  of  the  border  type 
windows  were  re-classified  with  100  percent  correct  feature 
classes.  This  means  that  the  feature  classes  that  RICA  selected 
were  complete  and  accurate. 

By  definition,  a  border  type  window  should  be  composed  of 
two  or  more  feature  classes,  but  in  16.5  percent  of  the  border 
type  windows  RICA  was  only  able  to  identify  one  of  these  feature 
classes.  In  all  of  these  cases,  the  feature  class  selected  was 
one  of  the  correct  classes,  and  also  was  the  class  that  comprised 
the  majority  of  the  area  in  the  window. 

The  reason  that  RICA  was  only  able  to  identify  one  of  the 
border  type  window's  feature  classes  was  due  to  the  fact  that 
there  was  not  enough  information  available  in  the  neighborhood  to 
make  a  reasonable  selection.  A  3  x  3  neighborhood  was  used  to 
help  determine  the  value  of  the  border  type  window.  In  the  case 
of  these  16.5  percent,  approximately  70  percent  of  the 
neighborhood  was  comprised  of  the  feature  class  that  RICA 
selected.  The  remaining  30  percent  was  divided  evenly  among  one 
or  more  of  the  other  feature  classes  without  a  strong  enough 
certainty  on  any  one  of  them  to  make  an  assertion  as  to  which 
one ( s )  made  up  up  the  remaining  portion  of  the  window. 

The  remaining  12.5  percent  of  the  correctly  re-classified 
border  type  windows  were  also  only  partially  re-classified,  but 
these  windows  contained  an  'unknown'  class  in  its  new  value.  In 


this  case,  as  in  the  previous  case,  those  feature  classes  that 
were  selected  along  with  the  'unknown'  value  were  correct,  but 
the  re-classification  value  was  incomplete.  This  occurred  when 
40  percent  or  more  of  a  window's  neighborhood  consisted  of  other 
border  type  or  'unknown'  windows. 

The  remaining  10.5  percent  of  the  total  border  type  windows 
were  re-classified  incorrectly.  In  all  cases,  one  of  the  feature 
classes  selected  by  RICA  was  correct  and  the  remaining  value(s) 
were  incorrect.  With  the  information  available,  RICA  made  a 
'reasonable'  re-classification  decision  and  had  more  information 
been  available  (such  as  the  direction  that  the  border  runs),  RICA 
could  have  been  able  to  make  more  accurate  decisions. 


4.  Analysis 

In  general,  RICA  was  successful  in  its  re-classification  of 
the  border  type  windows  in  the  Feature  Matrix.  With  the  limited 
amount  of  information  that  was  available  to  RICA,  it  was  able  to 
re-classify  to  some  degree  of  correctness  approximately  90 
percent  of  the  border  type  windows.  By  adding  the  proposed 
enhancements  of  a  learning  capability  and  neighborhood  expansion, 
and  by  adding  the  additional  information  of  various  image  feature 
data,  the  re-classification  done  by  RICA  should  become  more 
complete  and  more  accurate. 
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PRIMARY  APPLICATIONS  OF  Al 


1.  HEURISTIC  BORDER  SEARCH 

•  locate  all  border  windows 

•  arrange  border  windows  in  order  to  be  classified 


2.  Q-NN  CLASSIFICATION  RULE 

•  count  the  number  of  each  feature_class  in  neighborhood 

assign  <h>,  <m>,  <1> 

•  if  40%  of  neighborhood  *  BDR 

then  part  of  classification  =  unknown 


3.  RECLASSIFICATION 

•  frame-based  knowledge  representation  —  KE5  HT 

•  minimal  set  covers 


Analysis  of  Bayes  Classifier 
ETL  -vs-  SAE 


Image  UNF014  Number  Eficceolagg 

•  Windows  classified  the  same  242  94.5% 

•  Windows  classified  differently: 

•  Different  from  ETL  but  closer  to  actual  8  3.1% 

•  Different  from  ETL  but  not  closer  to  actual  6  2.4% 

256  100.0% 

Image  UNFQ26 

•  Windows  classified  the  same  242  94.5% 

•  Windows  classified  differently: 

•  Different  from  ETL  but  closer  to  actual  3  1 .2% 

•  Different  from  ETL  but  not  closer  to  actual  1 1  4.3% 

256  100.0% 
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RICH  module  Hierarchy 


Applications  of  Artificial  Intelligence 


Based  on  the  information  supplied  by  the  Initial  Classification  Module,  the 
following  is  an  analysis  of  the  Re-Classification  done  by  RICA. 


Image.  UNFOM 


Total  BDR  type  windows 

23 

9.0% 

100%  Correct  Classification 

65.0% 

*  Only  Partial  Classification  Determined 

17.0% 

*  With  unknown  as  Part  of  New  Classification 

9.0% 

Iotal  Correct 

91.0% 

Incorrect  Classifications 

9.0% 

Image  UNFQ26 

Total  BDR  type  Windows 

32 

12.5% 

100%  Correct  Classification 

56.0% 

*  Only  Partial  Classification  Determined 

16.0% 

*  With  unknown  as  Part  of  the  New  Classification 

16.0% 

SBSS*I 

Total  Correct 

88.0% 

Incorrect  Classifications 

12.0% 

*  Classifications  will  be  affected  by  RICA  enhancements 
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H  Vi  n  jypi 

\{  Vi  Vi  Vi  VI  VI  Vi  Vi  Vi  Vi  Vi  m|viNm[M 


Window  at  location  (1 1,2):  field  Window  at  location  (2,14):  water 

forest  forest 

Window  at  location  (3,6):  field  Window  at  location  (7,1 1 ):  water 

forest  field 

Window  at  location  (9,4):  field  Window  at  location  (6,12):  water 

forest 

Window  at  location  ( 1 0, 1 0):  water  Window  at  location  (5, 1 2):  water 

forest  forest 

Window  at  location  (8,5):  field  Window  at  location  (15,8):  water 

forest 

Window  at  location  (12,9):  water  Window  at  location  (14,8):  water 

forest  forest 

Window  at  location  (8, 1 1 ):  water  Window  at  location  ( 1 5,7):  field 

field  unknown 

Window  at  location  (9, 1 0):  water  Window  at  location  (1,11):  field 

forest  forest 

Window  at  location  ( 1 3,9):  water  Window  at  location  (1,12):  forest 

forest 

Window  at  location  ( 1 5,6):  field  Window  at  location  (1,15):  water 

forest  forest 

Window  at  location  (3,14):  water  Window  at  location  (16,7):  water 

forest 

unknown 


Window  at  location  (4, 1 3):  water 


PLANNED  ENHANCEMENTS 


1.  LEARNING 

2.  STRENGTH  OF  REGIONS  IN  BDR  WINDOW  NEIGHBORHOODS 

3.  PEAK  ANALYSIS  OF  THE  GRAY  SCALE  HISTOGRAM 
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APPENDIX  B 

A  Conceptual  Model  For  An 
Image  Understanding  System 


A  CONCEPTUAL  MODEL  FOR  AN  IMAGE  UNDERSTANDING  SYSTEM 


1 .  INTRODUCTION 

The  U.S.  Army  Engineer  Topographic  Laboratory  has  been  con¬ 
ducting  applied  research  in  the  area  of  terrain  feature  extrac¬ 
tion  from  aerial  radar  images.  The  work  has  focused  on  human 
analysis  and  the  experimental  application  of  image  processing 
algorithms.  Even  though  the  experimental  results  are  encourag¬ 
ing,  it  is  felt  that  heuristic  reasoning  from  the  realm  of 
artificial  intelligence  (AI)  used  in  a  complementary  manner  with 
the  image  processing  algorithms  can  result  in  more  powerful 
radar  image  understanding  automated  systems.  The  ETL  contracted 
with  Software  A&E,  Inc.,  to  analyze  existing  algorithms,  to 
determine  how  AI  techniques  can  be  employed  to  address  problem 
areas,  and  to  perform  experiments  that  illustrate  the  applica¬ 
tion  of  artificial  intelligence  to  the  image  understanding 
application. 

As  part  of  that  contract.  Software  ARE  was  tasked  to 
develop  and  outline  a  conceptual  model  for  a  radar  image  under¬ 
standing  system  that  would  embody  both  standard  image  processing 
techniques  and  AI  techniques.  This  technical  memorandum  docu¬ 
ments  the  required  conceptual  model. 

The  model  described  in  this  memorandum  derives  from  the 
programs  known  as  the  Radar  Image  Classification  Aid,  or  RICA, 
developed  for  the  task  in  the  Statement  of  Work  entitled  "Study 
of  Algorithms  and  Heuristics."  The  second  section  of  this 
memorandum  describes  a  conceptual  model  for  an  image  understand¬ 
ing  system  and  then  proceeds  to  outline  the  Radar  Image  Classif¬ 
ication  Aid  thus  showing  the  feasibility  of  the  conceptual 
model . 

The  system  described  by  this  model  is  aimed  initially  at 
the  image  understanding  researcher.  It  provides  a  flexible 
framework  into  which  new  techniques,  both  AI  and  non-AI,  can  be 
easily  integrated,  used  in  a  variety  of  combinations,  and 
evaluated.  That  same  flexibility  and  the  use  of  an  intelligent 
front-end  will  also  benefit  the  operational  user  because  it  will 
be  capable  of  applying  the  best  available  techniques  and  combi¬ 
nations  of  techniques  to  the  solution  of  a  variety  of  image 
understanding  problems. 


2. 


The  Model 


This  section  is  twofold  in  purpose.  First  a  conceptual 
model  for  an  image  understanding  system  is  outlined.  Then  fol¬ 
lowing  in  line  with  the  conceptual  model  the  experimental  image 
understanding  system  (RICA)  which  was  developed  is  presented. 

2.1  The  Conceptual  Model 

The  conceptual  model  for  an  image  understanding  system 
embodies  both  standard  image  processing  and  AI  techniques.  The 
model  is  designed  with  a  flexible  framework  allowing  the  image 
understanding  researcher  to  incorporate  new  AI  and  image  pro¬ 
cessing  techniques  into  the  system.  This  framework  also  allows 
the  efficient  evaluation  of  the  results  of  new  techniques,  alone 
or  in  combination  with  other  techniques  by  permitting  the  user 
or  the  intelligent  front-end  to  select  the  techniques  used. 

The  architecture  of  the  Conceptual  Model  consists  of  three 
levels  (see  figure  2.1-1).  At  the  top,  an  intelligent  front-end 
called  the  System  Control  Expert  System  governs  the  processing 
of  a  radar  image.  The  purpose  of  this  expert  system  is  to 
determine  the  'best'  combination  of  image  processing  and  AI 
techniques  to  apply  to  the  task  of  understanding  a  radar  image. 
This  expert  system  would  continually  interpret  results  received 
from  the  various  sub-processes  and  adjust  subsequent  processing 
to  account  for  interim  results.  In  order  to  keep  the  model  as 
flexible  as  possible  the  user  would  also  have  the  ability  to 
request  control  over  the  selection  of  the  combination  of  tech¬ 
niques  to  be  used. 

Below  the  System  Control  Expert  System  are  two  modules:  the 
Information  Gathering  Module,  and  the  Image  Reasoner  Module. 
The  purpose  of  the  Information  Gathering  Module  is  to  furnish 
detailed  information  about  the  image  to  the  system.  Two  types 
of  information  can  be  supplied:  Image  Processing  Information  and 
Contextual  Image  Information. 

Image  Processing  Information  consists  of  results  from  stan¬ 
dard  image  processing  techniques  such  as  Bayes  Theorem,  Histo¬ 
gram  analysis.  Hough  Transform,  etc.,  which  are  generally 
applied  to  an  entire  image.  Contextual  Image  Information  con¬ 
sists  of  algorithms  enhanced  with  heuristics  that  are  typically 
applied  to  a  specific  area  or  region  in  a  radar  image  but  that 
could  be  applied  to  an  entire  image. 

The  purpose  of  the  Image  Reasoner  Module  is  to  provide  the 
ability  to  'reason'  or  make  decisions  about  some  aspect  of  radar 
image  understanding.  This  module  consists  of  a  series  of  domain 
specific  expert  systems.  When  needed  these  expert  systems  would 
use  results  from  the  Information  Gathering  Modules  to  aid  in  the 


'reasoning'  process.  For  example,  in  the  RICA,  an  Image  Reclas¬ 
sification  Expert  System  has  the  ability  to  determine  the  com¬ 
ponents  of  windows  (  32  x  32  pixel  areas)  that  are  made  up  of  at 
least  two  of  the  following  feature  classes:  field,  forest, 
water,  city.  There  could  also  be  expert  systems  that  contain 
knowledge  in  domains  such  as  classifying  an  entire  image  for  a 
specific  geographic  location,  checking  the  feasibility  of  a 
classification  or  classifications  in  a  given  image  or  image 
type,  etc. 


CONCEPTUAL  MODEL  ARCHITECTURE 


Figure  2.1-1 


2.2  Radar  Image  Classification  Aid  (RICA) 


RICA  is  an  application  of  AI  techniques  to  the  area  of 
image  understanding.  RICA  divides  a  512x512  pixel  radar  image 
in  to  32x32  pixel  areas  called  windows.  Depending  on  its  con¬ 
tents  each  window  is  classified  as  one  of  the  following  feature 
classes:  field,  forest,  water,  city,  or  border.  A  border  window 
represents  an  area  consisting  of  at  least  two  of  the  remaining 
four  feature  classes.  The  aim  of  RICA  is  to  determine  the 
feature  classes  that  compose  each  of  the  border  type  windows  in 
a  given  radar  image.  In  general  RICA  reclassified  each  border 
type  window,  one  at  a  time,  by  requesting  various  types  of 
information  about  the  window  and  its  surrounding  area,  and  then 
reasoning  about  that  information. 

2.2.1  The  Architecture  of  RICA 

The  RICA  architecture  consists  of  several  levels,  with  each 
level  representing  varying  degrees  of  information  and  detail 
(see  figure  2. 2. 1-1).  At  the  top  level  of  the  RICA  architecture 
is  the  System  Control  Expert  System  (1.0).  This  module  provides 
the  RICA  user  with  the  ability  to  tell  the  system  the  various 
types  of  preliminary  processing  to  be  performed  on  an  image 
prior  to  reclassification  and  to  specify  which  of  the  available 
processing  options  should  be  used.  The  System  Control  Expert 
System  provides  a  basis  for  further  enhancements  in  which  a 
RICA  user  may  let  the  System  Control  Expert  System  'decide'  the 
best  means  of  processing  an  image. 

Below  the  System  Control  Expert  System  are  two  modules: 
the  Information  Gathering  Module  (1.1)  and  the  Image  Reasoner 
Module  (1.2).  The  purpose  of  the  Information  Gathering  Module 
is  to  provide  the  System  Control  Expert  System  with  various 
types  of  image  information.  This  module  consists  of  two 
modules:  the  Image  Processing  Information  Module  (1.1.1)  and 
the  Contextual  Image  Information  Module  (1.1.2). 

The  purpose  of  the  Image  Processing  Information  Module  is 
to  provide  standard  image  processing  types  of  algorithms  which 
are  generally  applied  to  an  entire  radar  image.  This  module 
currently  consists  of  the  Initial  Classification  Module 
(1.1. 1.1)  which  is  based  on  the  Bayes  Classifier  program  used  at 
the  ETL.  The  Initial  Classif icaton  Module  performs  the  task  of 
initially  assigning  one  of  the  feature  classes  (field,  forest, 
water,  city,  border)  to  each  window  in  a  given  image.  The 
result  of  this  module  is  a  16x16  Feature  Matrix  that  contains 
the  initial  classification  of  each  window  in  the  radar  image. 
The  Feature  Matrix  may  then  be  used  by  other  modules  in  the  sys¬ 
tem. 
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The  Contextual  Image  Information  Module  (1.1.2)  provides 
the  system  with  algorithms  enhanced  with  heuristics.  These 
algorithms  are  typically  applied  to  specific  regions  in  a  radar 
image  but  can  be  applied  to  an  entire  image.  This  module 
currently  consists  of  three  modules:  the  Border  Search  Module 
(1.1. 2.1),  the  Pixel  Level  Information  Module  (1.1. 2.2),  and  the 
Feature  Matrix  Level  Information  Module  (1.1.2. 3). 

The  Border  Search  Module  is  used  by  the  RICA  to  locate  and 
arrange  all  border  type  windows  in  the  order  in  which  they  are 
to  be  reclassified.  The  heuristic  used  in  this  module  is  that 
border  type  windows  with  the  most  lenown,  meaning  the  least 
number  of  unknown  or  border  type  windows  in  their  neighborhood 
(a  3  x  3  matrix)  are  candidates  to  be  reclassified  first.  Also, 
if  the  RICA  used  its  ability  to  'learn'  as  it  goes  through  the 
reclassif ication  process,  then  eventually  those  border  type  win¬ 
dows  that  originally  had  many  unknowns  in  their  neighborhoods 
will  have  been  updated  enough  to  reasonably  reclassify  them. 

The  Pixel  Level  Information  Module  provides  pixel  level 
contextual  information  to  the  RICA  system.  The  difference 
between  the  function  of  this  module  and  the  Image  Processing 
Module  (1.1.1)  is  that  this  module  does  image  processing  on  a 
selected  area  of  the  image  (a  window  for  instance)  and  the  Image 
Processing  Module  works  with  an  entire  image.  This  module 
currently  consists  of  two  modules:  the  'Next  Best'  Determinant 
Module  (1.1. 2. 2.1),  and  the  Analysis  of  Window  Quadrants  Module 
( 1.1. 2.2.2) . 

The  'Next  Best'  Determinant  Module  finds  the  next  best 
legal  feature  class  for  a  window.  By  definition,  the  best 
feature  class  would  be  the  current  feature  class.  This  classif¬ 
ication  would  also  correspond  to  the  highest  determinant. 
Therefore,  the  'Next  Best'  legal  determinant  would  describe  a 
feature  class  that  has  the  next  highest  determinant  value  and  is 
also  present  in  the  window's  neighborhood. 

The  Analysis  of  Window  Quadrants  Module  divides  a  border 
type  window  into  quadrants.  The  area  surrounding  each  quadrant 
is  then  analyzed  to  determine  the  most  reasonable  classification 
for  that  portion  of  the  border  window.  Therefore  each  border 
window  quadrant  is  analyzed  in  the  context  of  its  surrounding 
neighborhood. 

The  Feature  Matrix  Level  Information  Module  (1. 1.2.3)  pro¬ 
vides  abstract  Feature  Matrix  level  contextual  information  to 
the  RICA,  using  AI  and  image  processing  techniques. 

This  module  currently  consists  of  two  modules:  the  q-NN 
Classification  Rule  Module  (1.1. 2. 3.1),  and  the  Neighborhood 
Expansion  Module  (1.1. 2. 3. 2).  These  two  modules  actually  work 
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together.  The  q-NN  Classification  Rule  Module  examines  the  3x3 
neighborhood  of  a  given  border  type  window  and  calculates  the 
percentage  that  each  of  the  five  possible  feature  classes  com¬ 
poses  in  that  neighborhood.  This  module  uses  the  heuristic  that 
the  feature  class  with  the  highest  percentage  will  have  a  higher 
likelihood  of  composing  part  of  the  border  type  window. 

If  there  is  not  'sufficient  information'  found  in  the  ini¬ 
tial  3x3  neighborhood,  and  if  the  Neighborhood  Expansion  Module 
has  been  activated  by  the  System  Control  Expert  System 
(currently  this  must  be  indicated  by  the  user),  then  the  size  of 
the  neighborhood  will  be  expanded  until  a  reasonable  amount  of 
information  has  been  obtained.  In  this  case,  'sufficient  infor¬ 
mation'  means  that  less  than  50%  of  a  neighborhood  are  border 
type  windows.  With  each  expansion  iteration  the  q-NN  Classifi¬ 
cation  Rule  Module  is  used. 

The  second  module  under  the  System  Control  Expert  System  is 
the  Image  Reasoner  Module  (1.2).  The  task  of  this  module  is  to 
provide  the  facility  to  'reason'  about  different  aspects  of  a 
radar  image  using  various  domain  specific  expert  systems.  In 
the  current  RICA  system  there  is  only  one  domain  specific  expert 
system:  the  Image  Reclassification  Expert  System  (1.2.1), 
although  it  could  be  expanded  to  contain  any  number  of  expert 
systems . 

The  domain  of  the  Image  Reclassification  Expert  System  is 
the  reclassification  of  border  type  windows  in  a  Feature  Matrix. 
It  does  this  through  the  use  of  other  modules  and  sub-expert 
systems.  The  current  implementation  contains  a  Reclassification 
Expert  System  (1.2. 1.1),  and  a  'Learning'  Module  (1.2. 1.2). 

The  Reclassification  Expert  System  is  a  framed  based  system 
written  in  the  Software  A&E  Knowledge  Engineering  System  (KES), 
Hypothesize  and  Test  (HT)  subsystem  language  CKES843,  CREGG83D . 
Its  purpose  is  to  gather  information  about  a  given  border  type 
window,  provided  by  the  various  modules  under  the  Information 
Gathering  Module  (1.1),  and  determine  the  feature  classes  that 
compose  that  window.  It  does  not  do  the  actual  reclassification 
however,  but  simply  makes  a  decision  as  to  what  combination  of 
the  four  possible  feature  classes  best  fits  the  framed  based 
descriptions  for  each  of  the  feature  classes. 

The  'Learning'  Module  takes  the  Reclassification  Expert 
System's  decision  and  updates  the  Feature  Matrix,  making  that 
new  information  available  when  neighboring  windows  are  reclassi- 
f  ied. 


3. 


Conclusion 


This  technical  memorandum  has  described  a  conceptual  model 
for  an  image  understanding  system.  The  model  provides  an  intel¬ 
ligent  front-end  that  can  interpret  results  of  ongoing  process¬ 
ing  and  adjust  subsequent  processing  to  account  for  those 
interim  results.  The  architecture  of  the  model  is  such  that  new 
techniques  can  be  easily  added  and  evaluated.  The  model 
features  an  embedded  inference  mechanism  that  allows  the 
integration  and  evaluation  of  heuristic  techniques  based  upon 
knowledge  based  expert  system  technology  into  the  image  under¬ 
standing  system. 

Research  and  development  into  new  and  enhanced  image  under¬ 
standing  techniques  can  be  advanced  with  the  creation  and  use  of 
increasingly  powerful  tools.  The  system  described  in  this 
memorandum  is  one  such  tool,  a  test  bed  where  new  techniques  can 
be  prototyped  and  easily  integrated  with  other  techniques,  that 
can  provide  researchers  with  the  capability  to  efficiently 
evaluate  new  techniques  alone  and  in  various  combinations  with 
other  techniques.  Such  a  system  could  indeed,  naturally  evolve 
into  a  system  useful  in  an  operational  environment. 
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